Computing Discourse Information with Statistical Methods1~2

This dissertation research involves implementing a computer system that, given a natural language dialogue, will automatically tag each utterance with a discourse label (a concise abstraction of the intentional function of the speaker) and a discourse pointer (a focusing mechanism that represents the dialogue context in which an utterance is to be understood). (Samuel 1996) Since the discourse label of an utterance is dependent on the surrounding dialogue, tagging utterances with discourse labels is similar to the part-of-speech (PoS) tagging problem in syntax. Within the domain of PoS tagging, extensive experimental research has shown that statistical learning algorithms are among the most successful. I will investigate two methods that have been effective in PoS tagging: Hidden Markov Models (HMMs) (Ch arniak 1993) and TransformationBased Learning (TBL) (Brill 1995). Unlike these PoS taggers, which determine a word’s tag based on the surrounding words (within a fixed window size), a discourse-tagging system must use the surrounding utterances as input. Thus, the sparse data problem is much more severe for the discourse tagger, since the number of possible utterances is infinite. To alleviate this problem, rather than directly processing each utterance verbatim (which would probably bombard the system with a great deal of extraneous information that is not relevant to the task at hand), I have identified a small set of features that can be extracted from each utterance to provide the relevant information to the learning algorithm. Since HMMs and TBL deal with contiguous sequences of discourse labels, they are unable to take focus shifts into consideration. But it is crucial to account for the focus shifts that frequently occur in discourse. I have proposed a solution to this problem for both algorithms. For HMMs, this involves modifying the Markov assumption slightly, while still retaining the linear-time efficiency of the HMMs approach. With TBL, the solution is more straightforward.